Skip to content

server : disable speculative decoding for SWA models #13970

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jun 2, 2025
Merged

server : disable speculative decoding for SWA models #13970

merged 2 commits into from
Jun 2, 2025

Conversation

@ggerganov
Copy link
Member

@ggerganov ggerganov commented Jun 2, 2025 •
edited
Loading

ref #13963

To properly support his, we first need to fix this TODO:

llama.cpp/src/llama-kv-cache-unified-iswa.cpp

Lines 98 to 106 in 272df3f

llama_memory_state_ptr llama_kv_cache_unified_iswa::init_batch(const llama_batch & batch, uint32_t n_ubatch, bool embd_pooled, bool logits_all) {
GGML_UNUSED(embd_pooled);
// TODO: if we fail with split_simple, we should attempt different splitting strategies
// but to do that properly, we first have to refactor the batches to be more flexible
auto sbatch = llama_sbatch(batch, hparams.n_embd, true, logits_all);
std::vector<llama_ubatch> ubatches;

@ggerganov ggerganov mentioned this pull request Jun 2, 2025
Closed
@ggerganov ggerganov changed the title server : use swa-full fo draft context server : disable speculative decoding for SWA models Jun 2, 2025
@ggerganov ggerganov marked this pull request as ready for review June 2, 2025 18:07
@ggerganov ggerganov requested a review from ngxson as a code owner June 2, 2025 18:07
@ggerganov ggerganov merged commit 3637576 into master Jun 2, 2025
46 checks passed
@ggerganov ggerganov deleted the gg/server-spec-swa-clear branch June 2, 2025 18:34
furyhawk pushed a commit to furyhawk/llama.cpp that referenced this pull request Jun 6, 2025
@ggerganov @furyhawk
server : disable speculative decoding for SWA models (ggml-org#13970)
bcac573 
* server : use swa-full fo draft context

ggml-ci

* server : disable speculative decoding for SWA models
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ngxson ngxson Awaiting requested review from ngxson ngxson is a code owner

Assignees

No one assigned

Labels

examples server

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ggerganov